Close response body in SendEvent/HealthCheck to fix FD + memory leak (#15) - #16
Conversation
SendEvent discarded the http.Post response (`_, err := ...`) and never closed resp.Body. An unclosed/undrained body keeps the underlying TCP connection out of the keep-alive pool, leaking a connection + file descriptor on every call. SendEvent runs on every stats tick, so the agent's open-FD count and memory grow unbounded: long-lived agents hit "socket: too many open files" (event POSTs, then docker.sock stats fetches) and busy agents are OOM-killed (exit 137). Drain + close the body in SendEvent (and the matching http.Get in HealthCheck). Fixes swarmpit#15. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
I'm getting this problem too. Is reading the response body really necessary to fix this issue, or can we just close it? This article just recommends a |
|
I've deployed this branch here for testing: https://github.com/12joan/swarmpit-agent/pkgs/container/swarmpit-agent/1137758311?tag=fix-sendevent-response-body-leak |
|
@12joan close alone fixes the fd leak, so your instinct is right. the drain is what lets the connection be reused — verified against current master (which now carries the #736 nil-deref fix): build, vet and tests pass on golang:1.12. merging — thanks @michaeltoop for the fix and @12joan for confirming it in the field. nit, not blocking: |
Fixes #15.
Problem
SendEvent(swarmpit/client.go) discards thehttp.Postresponse and never closesresp.Body:An unclosed/undrained response body keeps the underlying TCP connection out of the keep-alive pool, so each call leaks a connection + file descriptor.
SendEventruns on every stats tick, so the agent's open-FD count and memory grow without bound.In production (7-node Swarm, agent as a
globalservice) this shows up as:socket: too many open files— first on the event POST (Post http://swarmpit:8080/events), then collaterally on the docker.sock stats fetches — after which the agent stays alive but stops shipping events (UI graphs go blank);exit 137,OOMKilled=true) on a roughly hourly cadence.The
ContainerUsagestats path already doesdefer resp.Body.Close(); onlySendEventwas missing it.Fix
Drain and close the body in
SendEvent(and the matchinghttp.GetinHealthCheck, which had the same omission):ioutil.Discardis used to stay compatible with the module'sgo 1.12(io.Discardlanded in 1.16). Builds clean withgo build ./swarmpit/.🤖 Generated with Claude Code